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METHOD OF PROCESSING IMAGE SIGNAL FROM SOLID-STATE 
IMAGING DEVICE, IMAGE SIGNAL PROCESSING APPARATUS, 
IMAGE SIGNAL GENERATING APPARATUS AND COMPUTER PROGRAM 
PRODUCT FOR IMAGE SIGNAL PROCESSING METHOD 

BACKGROUND OF THE INVENTION 

The present invention relates to a method of 
processing the color component signal (image signal) 
obtained from a solid-state image pickup device, an 
5 image signal processing apparatus, an image signal 
generating apparatus comprising the image signal 
processing apparatus and the solid-state image pickup 
device, and a computer program product for the image 
signal processing method. 

10 The solid-state image pickup device or the 

image sensor has a two-dimensional arrangement of a 
plurality of photoelectric elements such as 
photodiodes. For example, both the three primary color 
signals such as R, B and G signals and the luminance 

15 signal are produced in a single-chip solid-state image 
pickup device. A monochromatic color filter of R (red 
light penetrating), G (green light penetrating) or B 
(blue light penetrating) is arranged on the light- 
receiving surface of each photoelectric element. A 

20 white filter (a filter of white light penetrating) is 
arranged in place of the G filter in some cases. A 
plurality of photoelectric elements including these 
three types of color filters are arranged on the light- 



receiving surface in a predetermined pattern such as 
Bayer arrangement pattern. Hereinafter, we will 
explain examples wherein each photoelectric element 
corresponds to a pixel. The solid-state image pickup 
device is of either CCD type or MOS FET type. 

Generally, the color component signals 
composed of all the three primary colors are required 
from each pixel for producing high quality full-color 
image signals. In the single-chip solid-state image 
pickup device, however, only a monochromatic color 
component signal corresponding to the color of the 
filter of a respective photoelectric element of a pixel 
can be produced in each pixel. In each pixel of a 
given color in the single-chip solid-state image pickup 
device the two remaining color component signals are 
produced in such a manner that the signals from the 
adjacent pixels having different color filters are 
subjected to the signal processing such as 
interpolation. 

An example of the image signal processing 
method as described above is disclosed in US Patent No. 
5,382,976. This US patent describes a signal 
processing method for a single image sensor including a 
red pixel, a green pixel and a blue pixel. For the red 
pixel or the blue pixel not including the green 
component, the green value is obtained by interpolating 
the signal values of the pixels adjacent to the 
particular pixel. Also, the US patent includes the 



description to the effect that with regard to the green 
value of phase "00" containing the green value or phase 
"11"/ the adjacent pixel values are not interpolated 
but the signal values of the phases are used as they 
5 are without interpolation as indicated by G = G(0,0) in 
Figs. 4A and 4B of the U.S. patent. 

Now, a specific example of an image signal 
processing method will be explained with reference to 
the drawings . 

10 Fig. 2 is a diagram showing an example a 

block configuration of an image signal processing 
apparatus comprising a single-chip solid-state image 
pickup device and various processing circuits for 
processing the image signal picked up by the solid- 

15 state image pickup device. In Fig. 2, reference 

numeral 1 designates a solid-state image pickup device 
such as a charge-coupled device (CCD) image sensor for 
converting the imaging light into electric charge and 
outputting it as a video signal, numeral 2 a sample- 

20 hold and automatic gain control (CDS&AGC) circuit (CDS: 
Correlated Double Sampling) for sample holding the 
video signal outputted from the CCD 1 and outputting 
the video signal by amplifying the sample hold signal 
to a required level, and numeral 3 an A/D converter for 

25 converting the analog video signal from the CDS&AGC 
circuit 2 into a digital video signal. 

Numeral 4 designates a DSP (digital signal 
processor) circuit for processing the video signal from 



the A/D converter 3 as required. 

The color filters of the solid-state image 
pickup device (CCD) 1 are arranged in Bayer arrangement 
pattern. Fig. 3A shows the Bayer arrangement pattern 
5 and the arrangement of the pixel addresses attached to 
the pixels. In Fig. 3A, a green light transmitting 
filter (a filter of green light penetrating) is 
designated as G, a red light transmitting filter (a 
filter of red light penetrating) as R and a blue light 

10 transmitting filter (a filter of blue light 

penetrating) as B. Further, the number 1 is attached 
to the highest row and the leftmost column. For 
example, the pixel address of a pixel having a green 
light transmitting filter located on the mth row from 

15 top and the nth column from left is expressed as G m>n . 
The symbol of the pixel address is also expressed as 
the value of the color component signal obtained by 
picking up an image in the particular pixel. 

A method of image signal processing for the 

20 green component signal constituting one of the 

processes executed in the DSP circuit 4 described above 
will be explained with reference to Fig. 4C showing an 
example of the pixel interpolation method. The method 
(i) or (ii) described below is adaptively used for each 

25 pixel address. Specifically, in method (i) , as for the 
green component signal level of a pixel address (e.g. 
G 22 ) having a green light transmitting filter, the value 
of the green component signal level of the particular 



pixel address is used as it is. In method (ii), on the 
other hand, as for the green component signal level of 
a pixel address (e.g. R 25 ) having a red light 
transmitting filter or the green component signal level 
5 of a pixel address (e.g. B 34 ) having a blue light 
transmitting filter, the average value of the green 
component signal levels obtained from the pixels (e.g. 
G 13 , G 22 , G 24/ G 33 ) having a green light transmitting 
filter at the upper, lower, left and right adjacent 

10 pixel addresses is used. 

A specific example of the signal processing 
method (ii) described above will be explained in more 
detail. As shown in Fig. 4C, in the case where the 
image signal is processed for the pixel address R 23 to 

15 generate the green component signal level q 2 . by 

interpolation, for example, the value is calculated 
from 

g 23 = (G 13 + G 22 + G 24 + G 33 )/4 (1) 
Another method of calculation included in the 
20 signal processing method (ii) uses the values of only 
the left and right adjacent pixels of the pixel address 
R 23 as shown in equation (2) below for calculation. 

g 23 = (G 22 + G 24 )/2 (2) 
Various other alternative methods are conceivable 
25 including a method in which only the upper and lower 
adjacent pixel values or only the diagonally adjacent 
pixel values of the pixel address R 23 are used for 
calculation . 
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Fig. 4A is a diagram showing an interpolation 
method for the R component signal at the position of 
the pixel address G 22 . In this case, the red component 
signal at the position of G 22 has a pixel level equal to 
5 the average of the imaging red component signal levels 
of the pixel address R 21 and the pixel address R :i . Fig. 
4B is a diagram showing an interpolation method for the 
B component signal at the position of the pixel address 
G 22 . In this case, the blue component signal at the 

10 position of G 22 has a pixel level equal to the average 
of the imaging blue component signal levels of the 
pixel address B 12 and the pixel address B 32 . Further, 
Fig. 4D is a diagram showing an interpolation method 
for the B component signal at the position of the pixel 

15 address R 23 . In this case, the blue component signal at 
the position of R 23 has a pixel level equal to the 
average of the imaging blue component signal levels at 
the respective positions of the pixel addresses B 12 , B 14 , 
B 32 and B 34 . 

20 The signal processing method described above 

has the following problem. Specifically, assume that 
the band limiting range of the source follower circuit 
of the amplifier 15 of the output unit of the CCD used 
for securing the required S/N ratio, the bandwidth of 

25 the CDS circuit 2 or the band limiting range of the 
process amplifier is excessively narrow. Then, the 
waveform of the output signal is distorted in each 
case. As a result, a horizontal line having the blue 



light transmitting filter such as the (m-l)th line 
(row) in Fig. 3A and the horizontal line having the red 
light transmitting filter such as the mth line (row) in 
Fig. 3A have different waveform distortion amounts of 
5 the green component signal. Thus, the error levels 
applied to the green component signals are different 
between adjacent lines, often resulting in variations 
of the green component signal level, even when a 
uniform green component light in the imaging light is 

10 applied from the object to each pixel. The green 
component signals varied from one line to another 
causes horizontal stripes of noises on the image 
subjected to the signal processing as shown in Fig. 4C. 

Also, in the case where the conditions for 

15 the production process undergo a change, the filter 
characteristic of the green light transmitting filter 
among the CCD color filters may be varied in every 
other horizontal scanning line, i.e. between a 
horizontal line having the blue light transmitting 

20 filter and a horizontal line having the red light 

transmitting filter, which often causes variations of 
the green component signal level between different 
lines . 

This often leads to the problem that even 
25 when a uniform green component light in the imaging 
light is applied from an object to each pixel, the 
levels of the green component signals corresponding to 
the pixels of the green light transmitting filters 
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outputted from the CCD may be varied between even-row 
horizontal lines and odd-row horizontal lines. 

Figs. 6A and 6B show waveforms obtained by 
sampling the output signal of the CCD in a correlated 
5 double sampling (CDS) circuit when the CCD having the 
aforementioned problem picks up an object having a 
uniform brightness and color in a whole picture frame. 
Fig. 6A shows a signal waveform of a horizontal line 
(the mth horizontal line in Fig. 3A, for example) 
10 having a red light transmitting filter and green one, 
and Fig. 6B shows a signal waveform of a horizontal 
line (the (m-l)th horizontal line in Fig. 3A, for 
example) having a blue light transmitting filter and 
green one . 

15 in Figs. 6A and 6B, numerals 100, 200 

designate reference voltage signal (for example, an 
optical black signal) waveforms. Numeral 110 
designates an image signal waveform corresponding to 
the reference voltage signal waveform 100, and numeral 

20 210 designates an image signal waveform corresponding 
to the reference voltage signal waveform 200. The 
values of the color component signals of the image 
signal waveforms 110, 210 are indicated by the level 
difference with the reference voltage signal waveforms 

25 100, 200, respectively. In the waveform 110, from left 
to right, the level waveforms of the red component 
signal, the green component signal and the red 
component signal are shown, while in the waveform 210, 



from left to right, the level waveforms of the green 
component signal, the blue component signal and the 
green component signal are shown. 

Comparison between the green component signal 
5 of the waveform 110 in Fig. 6A and the green component 
signal of the waveform 210 in Fig. 6B shows that the 
signal levels are different between them due to the 
problem described above. In the case where an image is 
displayed on a monitor (not shown) inserted in a 

10 subsequent stage, this level difference is seen to 

cause a level difference between horizontal lines even 
in a displayed image produced by imaging an object 
uniformly. Thus the image is displayed in horizontal 
stripes or speckles, thereby extremely deteriorating 

15 the image quality. 

SUMMARY OF THE INVENTION 

The object of the present invention is to 
provide means for preventing the unnecessary level 
difference from being caused between horizontal lines 

20 or reducing the unnecessary level difference even in 
the case where the picking up characteristics of the 
CCD are varied or the performance thereof is 
deteriorated and thereby to prevent the deterioration 
of the image quality which otherwise might be caused by 

25 horizontal stripes or speckles on a picked-up image or 
a part of image by the imaging light from the object. 

According to one aspect of the invention, 



there is provided an image signal processing method for 
processing the color component signals obtained from a 
solid-state image pickup device including an 
arrangement of a plurality of photoelectric elements 
5 with color filters arranged in the light receiving 
section of the photoelectric elements, comprising the 
steps of fetching a color component signal from a 
designated pixel corresponding to the photoelectric 
element located on a line and having a filter that can 
10 transmit the green light in the solid-state image 
pickup device, fetching the color component signals 
from the pixel corresponding to the photoelectric 
element on other pixel line than the first line of the 
designated pixel in the neighborhood of the designated 
15 pixel, the neighboring pixels each having a filter for 
transmitting at least the green light, and determining 
the value of the color component signal of the 
designated pixel corresponding to the photoelectric 
element having the filter that can transmit at least 
20 the green light, based on a plurality of the color 
signals fetched in the foregoing steps. 

According to another aspect of the invention, 
there is provided an image signal processing apparatus 
for processing the color component signals obtained by 
25 a solid - state image pickup device including an 

arrangement of a plurality of photoelectric elements 
and a color filter arranged in the light receiving 
section of each pixel corresponding to the 



photoelectric element comprising means for fetching the 
color component signal from a designated pixel 
corresponding to the photoelectric element having a 
filter that can transmit at least the green light on 
5 the lines of the solid-state image pickup device, means 
for fetching the color component signal from a pixel 
corresponding to the photoelectric element on other 
pixel line than said line of the designated pixel in 
the neighborhood of the designated pixel, the 

10 neighboring pixels each having a filter for 

transmitting at least the green light, and arithmetic 
means for determining the value of the color component 
signal of the designated pixel corresponding to the 
photoelectric element having the filter that can at 

15 least the green light, based on a plurality of the 
color signals fetched. 

According to still another aspect of the 
invention, there is provided an image signal generating 
apparatus comprising a solid-state image pickup device 

20 including an arrangement of a plurality of 

photoelectric elements and a color filter arranged in 
the light receiving section of each pixel corresponding 
to the photoelectric element, and an image signal 
processing unit for processing the color component 

25 signals obtained by the solid-state image pickup 

device, wherein the signal processing unit includes 
means for fetching the color component signal from a 
designated pixel corresponding to the photoelectric 



element located on a line of the solid-state image 
pickup device and having a filter that can transmit at 
least the green light, means for fetching the color 
component signals from the pixel corresponding to the 
5 photoelectric element located on other lines in the 
neighborhood of the designated pixel, the neighboring 
pixels each having a filter capable of transmitting at 
least the green light, and arithmetic means for 
determining the value of the color component signal of 

10 the designated pixel corresponding to the photoelectric 
element having a filter capable of transmitting at 
least the green light, based on a plurality of the 
color signals fetched. 

According to a further aspect of the 

15 invention, there is provided a computer program product 
comprising a computer usable medium having computer 
readable program code means embodied therein for 
processing the color component signals obtained by a 
solid-state image pickup device having an arrangement 

20 of a plurality of photoelectric elements with a color 
filter arranged in the light receiving section of each 
of the pixels corresponding to the photoelectric 
elements, wherein the computer readable program code 
means includes means for fetching the color component 

25 signal from a designated pixel corresponding to the 
photoelectric element located on a line of the solid- 
state image pickup device and having a filter that can 
pass at least the green light, means for fetching the 
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color component signals from the pixel corresponding to 
the photoelectric element in the neighborhood of the 
designated pixel and located on line different from the 
designated pixel, the neighboring pixels being located 
5 on lines and each having a filter capable of 

transmitting at least the green light, and means for 
determining the value of the color component signal of 
the designated pixel corresponding to the photoelectric 
element having a filter capable of passing at least the 
10 green light, based on a plurality of the color signals 
fetched. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Figs. 1A and IB are diagrams for explaining 
an example of an image signal processing method 
15 according to the invention. 

Fig. 2 is a diagram showing an example block 
configuration of an image signal processing apparatus. 

Figs. 3A and 3B are diagrams showing the 
Bayer arrangement pattern and another arrangement of 
20 filters, i.e. pixel addresses attached to the pixels. 

Figs. 4A, 4B, 4C and 4D are diagrams for 
explaining an example of the pixel interpolation 
method. 

Figs. 5A and 5B are diagrams showing the 
25 level of the green component signal of each pixel for 
the image signal processing method according to the 
invention. 



Figs. 6A and 6B are diagrams showing 
waveforms for horizontal lines containing the pixels R 
and B lines sampled from the CCD output signal in a 
correlated double sampling circuit. 

Fig. 7 is a block diagram showing an image 
signal generating apparatus including a signal 
processing apparatus according to an embodiment of the 
invention . 

Fig. 8 is a block diagram showing an image 
signal generating apparatus including a signal 
processing apparatus according to another embodiment of 
the invention. 

Fig. 9 shows an example of the main flowchart 
for interpolation of the green component signal in a 
signal processing apparatus according to the invention. 

Fig. 10 shows an example of the flowchart for 
the signal processing performed in a signal processing 
apparatus according to this invention as shown in Fig. 
1A or Fig. IB. 

Fig. 11 shows an example of the flowchart for 
the signal processing performed by a signal processing 
apparatus according to this invention even in the case 
of a filter arrangement other than the Bayer 
arrangement pattern shown in Fig. 3B. 

Fig. 12 is a flowchart schematically showing 
the signal processing steps for a signal processing 
apparatus according to the invention. 



DESCRIPTION OF THE EMBODIMENTS 

Embodiments of the invention will be 
explained below. A block configuration of an image 
signal processing apparatus according to the invention 
5 is shown in Fig. 7. In Fig. 7, the same reference 
numerals as those in Fig. 2 designate the same 
component elements as the corresponding component 
elements, respectively, in Fig. 2. Also, the color 
filters of a CCD 1 are arranged in a manner similar to 

10 the Bayer arrangement pattern shown in Fig. 3A, and the 
indication of the pixel addresses and the symbol of a 
pixel address is expressed as the value of the color 
component signal obtained by picking up an image in the 
particular pixel. 

15 An image signal processing method, i.e. a 

method of processing the green component signal which 
is one of the processes carried out in a DSP circuit 40 
according to the invention will be explained with 
reference to Figs. 1A, IB and the flowcharts of Figs. 9 

20 to 12 showing an example of the image signal processing 
method according to the invention. 

Fig. 9 shows the main routine of the image 
signal processing in the DSP 40 of a signal processing 
apparatus according to an embodiment of the invention. 

25 First, in step 101, the video signals of all the pixel 
addresses of an image frame to be interpolated are 
stored in a frame buffer memory 41 from an A/D 
converter 3. As an alternative, the video signal of a 



pixel address G ffi(I1 corresponding to a green light 
transmitting filter of the image frame is stored in the 
frame buffer memory 41. 

In step 102, an interpolation process is 
5 carried out in an interpolation processing unit 42 for 
determining the green component value among the three 
color component signal values at the respective 
positions of the green pixel, the red pixel and the 
blue pixel. Hereinafter, the green pixel means a pixel 

10 corresponding to the photoelectric element provided 

with a color filter transmitting at least green light, 
the red pixel means a pixel corresponding to the 
photoelectric element provided with a color filter 
transmitting red light, and the blue pixel means a 

15 pixel corresponding to the photoelectric element 

provided with a color filter transmitting blue light. 
The green pixel may be replaced by a white pixel having 
a white filter which transmits green light and other 
particular color component light. A specific flow of 

20 the process performed in step 102 will be explained 
with reference to the interpolation processing flows 
shown in Figs. 10 and 11. In step 103, it is 
determined whether the interpolation processing is 
complete for all the pixel addresses in an image frame 

25 intended for interpolation, and the process of steps 
102 and 103 is repeated until the value of the green 
component is determined for all the pixel addresses. 

Now, a specific subroutine of step 102 will 



be explained. 

First, in the interpolation method shown in 
Fig. 1A, the signal value of the green pixel is 
determined by method (i') described below in accordance 
5 with the pixel address. Further, method (ii) is 
adaptively applied for each of red and blue pixel 
addresses . 

(i ? ) The green component signal of a pixel 
address G m , n having a green light transmitting filter is 

10 the average of the value of the green component signal 
obtained for the particular pixel address G ra , n and the 
value of the green component signal picked up by the 
pixel of the pixel address G^^^ on the immediately 
left column on the immediately upper row than the pixel 

15 address G m/n . 

In a specific example of application of 
method (i T ) described above, as shown in Fig. 1A, 
consider the case in which a green component signal 
level g 22 is obtained by the processing the image signal 

2 0 for the pixel address G 22 . The value is calculated from 
g 22 = (G u + G 22 ) /2 (3) 
This equation can be rewritten into a general 
expression below. 

g mn = (G m -i,n-i + G m , n )/2 (4) 

25 In the method shown in Fig. IB, on the other 

hand, the method (i" ) described below is carried out in 
accordance with the pixel address followed by 
adaptively carrying out the method (ii) described 
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above . 

(i") The green component signal of a pixel 
address G m>n having a green light transmitting filter is 
a weighted average of the value of the green component 
5 signal picked up in the particular pixel and the values 
of the green component signals obtained in the pixels 
having a green light transmitting filter at the four 
pixel addresses G m _ liU _ ir G m _ ljn+1 , G m+lrTl „ ir G m+ljn+1 diagonally 
upper and lower than the particular pixel address G m( „. 
10 As a specific example, as shown in Fig. IB, 

for example, assume that the green component signal 
level g 22 is to be obtained by image signal processing 
for the pixel address G 22 . The value is calculated as 
shown below, 

15 g 22 = (G n + G 13 + G 22 + G 31 + G 33 ) /5 (5) 

or by weighting pixel address G 22 more than other pixel 
addresses as shown below. 

g 22 = (G n + G 13 + 4G 22 + G 31 + G 33 )/8 (6) 
Equations (5) and (6) can be rewritten into general 

20 expressions shown below. 

g mn = (aG^n.! + 0G m _ 1/n+1 + 7G m<n + SG^,^ + 
£ G m+1 , n+1 ) /CO, (7) 
where co=a + j3 + r+£. 



Fig. 10 is a flowchart for the aforementioned 
25 interpolation processing method according to an 

embodiment. This flow is a flow of the interpolation 



processing of step 102 in the main flow shown in Fig. 
9. First, in step 201, the pixel address of the pixel 
intended for interpolation is designated. Assume that 
the designated pixel address is G mfn (m, n: positive 
5 integers) . In step 202, it is determined whether the 
designated pixel address is that of the pixel having a 
green (white) light transmitting filter or not. In the 
case where the designated pixel is not a green (white) 
pixel, the pixel data is read from the frame memory 41 
10 and the data values of the red and blue pixels are 
determined by equation (1), (2) or (2 T ) in step 203. 
In the case where the designated pixel is a green 
(white) pixel, on the other hand, the pixel data are 
read from the frame memory 41 and the data value of the 
15 green pixel is determined according to equation (4) or 
(7) in step 204. 

Various other calculation methods than the 
methods (i 1 ) and (i") are conceivable for the image 
processing method according to the invention. 
20 In the pattern of Fig. 1A, for example, the 

green signal value of the designated pixel address G mfn 
may be determined using the average value of the upper 
right pixel address G m _ ljn+1 and the lower left pixel 
address G^,^ each adjacent to the pixel address G m , n . 
25 Also, the green pixel on a horizontal line other than 
the upper and lower adjacent horizontal lines can be 
used as the green signal value of the designated pixel 
address G m/n . 
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The signal processing method and apparatus 
according to the invention described above are not 
limited to the image pickup device having the Bayer 
arrangement pattern, but is also applicable to other 
5 patterns of arrangement. For example, Fig. 11 shows a 
flowchart of interpolation for the color filter 
arrangement pattern shown in Fig. 3B. This flow is a 
routine in step 102 of the main flow shown in Fig. 9. 
First, the pixel address of the pixel 

10 intended for interpolation is designated in step 301. 
Assume that the designated pixel address is G m , R (m, n: 
positive integers) . In step 302, it is determined 
whether the designated pixel address is that of a pixel 
having a green (white) light transmitting filter or 

15 not. In the case where the designated pixel is not a 
green (white) one, the pixel data is read out from the 
frame memory 41 and the data values of the red and blue 
pixels are determined according to equation (2) in step 
303. In the case the designated pixel is a green 

20 (white) pixel, on the other hand, the pixel data is 

read out from the frame memory 41 and the data value of 
the green pixel is determined in accordance with 
equation (4) or (8) in step 304. Here, equation (8) is 
disclosed below. 



25 



g mn = (aG m _ 2 , n + $ G m _ 1(n + 7G m , n + <5G m+1/n 

+ £G m+2/n )/C0 (8) 



, where co = a+j3 + r+ <5 + £. 

Fig. 12 is a flowchart showing the 
interpolation processing of step 102 in Fig. 9, i.e. 
the flow of Figs. 10 and 11 in broader term. Steps 
5 401, 402 are similar to steps 201, 202, respectively, 
in Fig. 10. In step 403, the green component signal 
value g mn in the red pixel and the green component 
signal value g mn in the blue pixel are determined using 
the green (white) video signal component value obtained 

10 from the pixel of a green (white) light transmitting 
filter among those pixels adjacent to the pixel of the 
designated pixel address or the average of the 
predetermined green (white) component signal value. 
Further, in step 4 04, the green component signal value 

15 g mn of the green pixel is determined from the average of 
the green (white) component signal value obtained from 
the pixel of the designated pixel address and the green 
(white) component signal value obtained from at least 
one pixel of green (white) light transmitting filter 

20 among those pixels adjacent to the particular 
designated pixel. 

The behavior of the image signal level in the 
case of the image processing is conducted using an 
example of the image processing method according to the 

2 5 invention described above will be explained with 

reference to Figs. 5A and 5B. Figs. 5A and 5B show the 
level of the green component signal of each pixel while 
omitting the level of the red component signal and the 



blue component signal. 

Figs. 5A and 5B are diagrams showing, in a 
matrix for each pixel address, the behavior of the 
green component signal (image signal) before and after 
5 the processing of an image picked up from a uniform 
object using the method of Fig. 1A according to the 
invention. Fig. 5A shows the image signal level picked 
up before the image processing according to the 
invention, and Fig. 5B shows the image signal level 

10 obtained as the result of the image processing 
according to the invention. 

Fig. 5A shows the manner in which the green 
component signal level assumes different values of 140 
and 100 between odd rows, i.e. horizontal lines having 

15 a blue light transmitting filter and even rows, i.e. 
horizontal lines having a red light transmitting 
filter. In contrast, Fig. 5B shows the manner in which 
the image signal processing is conducted so that the 
green component signal level value is 12 0 for both odd 

20 rows, i.e. the horizontal lines having a blue light 

transmitting filter and even rows, i.e. the horizontal 
lines having a red light transmitting filter. 

The reason why the value of the green 
component signal level is 120 for both cases is that 

25 the image signal processing by the use of the method 
shown in Fig. 1A produces a green component signal 
level value 120 obtained by an average of 140 and 100 
which are signal levels before the interpolation. 



Also, as for the values of the pixels having either a 
red light transmitting filter or a blue light 
transmitting filter, the green component signal level 
value assumes 120 as the result of using the 
5 aforementioned interpolation technique. 

The foregoing description of embodiments 
refers to the case in which the image signal processing 
according to the invention is performed in the DSP 
circuit 40. Nevertheless, the method according to the 

10 invention can be implemented in other than the DSP 
circuit 40. As shown in Fig. 8, for example, a 
personal computer 43 in which a picked up image signal 
is input can perform the image processing according to 
the invention and records the processing result or 

15 outputs the processed signal to the stages subsequent 
to the personal computer. 

The flowcharts shown in Figs. 9, 10, 11 and 
12 according to embodiments of the invention can be 
stored in any of various media (not shown) such as a 

20 magnetic disk, an optical disk and a semiconductor 
memory in the form of a program code readable by the 
computer or the microprocessor, and the program is read 
from the recording medium by the DSP 40 or the personal 
computer 43 thereby to execute the steps of the 

25 flowchart. Also, the program code can be downloaded to 
the DSP 40 or the personal computer 43 through a 
communication channel {not shown) to carry out the 
signal processing method according to the invention. 



It will thus be understood from the foregoing 
description that according to this invention, even in 
the case where the green component signal level is 
different between odd and even horizontal scanning 
lines due to the variations of the CCD characteristics 
or the performance deterioration thereof, the image 
quality deterioration can be prevented and a clear 
image can be produced without causing a horizontal 
striped pattern or speckles on a picked-up image or a 
part of image by the imaging light from the object by 
preventing the unnecessary level difference from 
occurring between horizontal lines or reducing such 
level difference. 
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CLAIMS : 

1. An image signal processing method for 
processing a color component signal obtained by a 
solid-state image pickup device including an 
arrangement of a plurality of a photoelectric elements 
and a color filter arranged in the light receiving 
section of each of the pixels corresponding to the 
photoelectric elements, comprising the steps of: 

storing a first color component signal from a 
designated pixel corresponding to the photoelectric 
element having a filter capable of passing at least 
green light on a line of said solid-state image pickup 
device in a memory device ; 

storing a second color component signal from 
at least one pixel in the neighborhood of said 
designated pixel corresponding to said photoelectric 
element in said memory device, the neighboring pixel 
having a filter for transmitting at least the green 
light on a line different from said line; and 

interpolating the value of said first color 
component signal based on the value of said second 
color component signal in an interpolation processing 
unit . 

2. A method according to Claim 1, wherein said 
designated pixel is a pixel corresponding to the 
photoelectric element on said horizontal line, and the 
pixel in the neighborhood of said designated pixel 
includes a pixel on another horizontal line adjacent to 
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said designated pixel. 

3. A method according to Claim 2, wherein the 
average of said first color component signal of said 
designated pixel and said second color component signal 
of the pixel in the neighborhood of said designated 
pixel is calculated, and the value of said first color 
component signal is interpolated based on said average 
of said first and second color component signals. 

4. A method according to Claim 2, wherein said 
solid-state image pickup device includes a Bayer 
arrangement pattern having pixels corresponding to the 
photoelectric element with a red light pass filter, a 
pixel corresponding to the photoelectric element with a 
filter capable of transmitting at least the green light 
and a pixel corresponding to the photoelectric element 
with a blue light transmitting filter, and in the case 
where each of said pixels can be specified by the row 
number m of a horizontal line and the column number n 
of vertical line orthogonal to said horizontal line of 
said solid-state image pickup device (m, n: arbitrary 
positive integer) , assuming that the color signal 
component of said designated pixel having a filter 
capable of transmitting at least the green light on 
said horizontal line is given as G m , n , the value of the 
color component signal of said designated pixel having 
a filter capable of transmitting at least the green 
light is determined by at least selected one of the 
formulae (G m _ lrn _! + G m , n )/2, (G m+1 , n+1 + G m , n )/2, (G^,^ + 
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G m , n )/2 and (G^,^ + G m , n )/2. 

5. A method according to Claim 1, further 

comprising the step of determining the value of the 
green signal component in a pixel corresponding to the 
photoelectric element having a filter other than the 
one transmitting at least the green light using the 
value of the color component signal determined for said 
designated pixel corresponding to the photoelectric 
element having a filter transmitting at least the green 

An image signal processing apparatus for 
processing a color component signal obtained by a 
solid-state image pickup device including an 
arrangement of a plurality of photoelectric elements 
and a color filter arranged in the light receiving 
section of each of the pixels corresponding to the 
photoelectric element, comprising: 

a memory for storing a first color component 
signal from a designated pixel corresponding to the 
photoelectric element having a filter capable of 
transmitting at least a green light on a line of said 
solid-state image pickup device, said memory storing a 
second color component signal from a pixel 
corresponding to the photoelectric element in the 
neighborhood of said designated pixel, said neighboring 
pixel having a filter capable of transmitting at least 
a green light on a line different from said designated 
line; and 
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an interpolation processing unit for 
interpolating the value of said first color component 
signal based on the value of said second color 
component signal. 

7. An apparatus according to Claim 6, wherein 
said designated pixel is a pixel corresponding to the 
photoelectric element on said horizontal line, and said 
pixel in the neighborhood of said designated pixel 
includes a pixel on another horizontal line adjacent to 
said designated pixel. 

8. An apparatus according to Claim 7, wherein 
said interpolation processing unit interpolates the 
value of said first color component signal based on 
average value of said first color component signal of 
said designated pixel and said second color component 
signal of said pixel in the neighborhood of said 
designated pixel. 

9. An apparatus according to Claim 7, wherein in 
the case where said solid-state image pickup device has 
a Bayer arrangement pattern of pixels corresponding to 
the photoelectric element with a red light transmitting 
filter, a pixel corresponding to the photoelectric 
element with a filter capable of transmitting at least 
the green light and a pixel corresponding to the 
photoelectric element with a blue light transmitting 
filter, and each of said pixels can be specified by the 
row number m of a horizontal line and the column number 
n of the vertical line orthogonal to said horizontal 
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line of said solid-state image pickup device (m, n: 
arbitrary positive integer) , assuming that the color 
signal component of said designated pixel corresponding 
to the photoelectric element having a filter capable of 
transmitting at least the green light on a horizontal 
line is given as G mrn/ said arithmetic means determines 
the value of the color component signal of said 
designated pixel having a filter capable of 
transmitting at least the green light by at least 
selected one of the equations (G^-,,^ + G m<ri )/2, (G m+1/n+1 + 
G m#n )/2, (G m _ lfn+1 + G m ,J/2 and (G^,^ + G m , n )/2. 
10. An apparatus according to Claim 6, wherein 

said arithmetic means further includes means for 
determining the value of the green signal component in 
said pixel having the filter other than the filter for 
transmitting at least the green light using the value 
of the color component signal determined for said 
designated pixel having the filter for transmitting at 
least the green light. 



comprising a solid-state image pickup device including 
an arrangement of a plurality of photoelectric elements 
and a plurality of color filters arranged in the light 
receiving sections of the pixels corresponding to the 
photoelectric element, respectively, and an image 
signal processing unit for processing the color 
component signal obtained by said solid-state image 
pickup device, wherein said signal processing unit 




An image signal generating apparatus 
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includes : 



a memory for storing a first color component 



signal from a designated pixel corresponding to the 
photoelectric element having a filter capable of 
transmitting at least a green light on a line of said 
solid-state image pickup device, said memory storing a 
second color component signal from a pixel 
corresponding to the photoelectric element in the 
neighborhood of said designated pixel, said neighboring 
pixel having a filter for transmitting at least a green 
light on a line different from said designated line; 
and 



interpolating the value of said first color component 
signal based on the value of said second color 
component signal. 



readable program code means embodied therein for 
processing the color component signal obtained by a 
solid-state image pickup device having an arrangement 
of a plurality of photoelectric elements and a color 
filter arranged in the light receiving section of each 
of said pixels corresponding to the photoelectric 
element, said computer readable program code means 
comprising : 

means for storing a first color component 
signal from a designated pixel corresponding to the 



an interpolation processing unit for 




A computer program product comprising: 



a computer usable medium having computer 
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photoelectric element having a filter capable of 
passing the green light on a line of said solid-state 
image pickup device in a memory device; 

means for storing a second color component 
signal from at least one pixel in the neighborhood of 
said designated pixel corresponding to said 
photoelectric element in said memory device, the 
neighboring pixel corresponding to the photoelectric 
element having a filter for transmitting at least the 
green light on a line different from said designated 
pixel line; and 

means for interpolating the value of said 
first color component signal based on the value of said 
color component signal in an interpolation processing 
unit . 



\ 
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ABSTRACT OF THE DISCLOSURE 

An image signal processing method and an 
image signal processing apparatus for processing the 
color component signal obtained from a solid-state 
image pickup device including an arrangement of a 
plurality of photoelectric elements and a color filter 
arranged in the light receiving section of each pixel 
corresponding to the photoelectric element, an image 
signal generating apparatus and a computer program 
product for the image signal processing method. A 
color component signal is fetched from a designated 
pixel corresponding to the photoelectric element having 
a filter that can transmit the green light on a line of 
the solid-state image pickup device. A color component 
signal is also fetched from a pixel corresponding to 
the photoelectric element having a filter for 
transmitting at least the green light on another line, 
which pixel is located in the neighborhood of the 
designated pixel. Further, based on a plurality of the 
color signals fetched in the foregoing steps, the value 
of the color component signal of the designated pixel 
having the filter for transmitting at least the green 
light is determined. 
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101 



RECORD VIDEO SIGNALS OF ALL PIXEL ADDRESSES 
OF IMAGE MATRIX (FRAME) INTENDED FOR 
INTERPOLATION IN FRAME BUFFER MEMORY, 
OR 

RECORD VIDEO SIGNAL OF PIXEL ADDRESS 
CORRESPONDING TO GREEN LIGHT TRANSMITTING 
FILTER OF IMAGE MATRIX (FRAME) IN FRAME 
BUFFERMEMORY 
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^> I 

INTERPOLATION PROCESSING 
(FIGS. 10-12) 




FIG. 11 
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DESIGNATE PIXEL ADDRESS OF PIXEL TO BE 
INTERPOLATED. SET DESIGNATED PIXEL 
ADDRESS AS G m n 
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DESIGNATE PIXEL ADDRESS OF PIXEL TO BE 
INTERPOLATED. SET DESIGNATED PIXEL 
ADDRESS ASG mn 
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DESIGNATED PIXEL ^ 
ADDRESS CORRESPONDS TO GREEN 
(WHITE) LIGHT TRANSMITTING 
FILTER ? 
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403 



9mn 0R g m n = 

GREEN (WHITE) VIDEO SIGNAL COMPONENT VALUE 
OBTAINED FROM PIXEL OF GREEN (WHITE) LIGHT 
TRANSMITTING FILTER AMONG THE PIXELS ADJACENT 
TO PIXEL OF DESIGNATED PIXEL ADDRESS, 
OR AVERAGE VALUE OF PREDETERMINED GREEN 
(WHITE) VIDEO SIGNAL COMPONENT VALUES 



9mn 

AVERAGE VALUE OF GREEN (WHITE) VIDEO SIGNAL 

COMPONENT VALUE OBTAINED FROM PIXEL OF 
DESIGNATED PIXEL ADDRESS AND GREEN (WHITE) 
VIDEO SIGNAL COMPONENT VALUE OBTAINED 
FROM PIXEL WITH A FILTER TRANSMITTING AT 
LEAST GREEN (WHITE) LIGHT AMONG THOSE 
PIXELS ADJACENT TO DESIGNATED PIXEL 
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Japanese Language Declaration 

(B*itft) 



mt, *ffl»f 3 5 481 1 9* (a) -(d) mxm 6 5$(b) 
v^SWffFtS**»3 6 5 (a) gCMS-f <B&ttiB. xrt*l-HT? 

Prior Foreign Application(s) 

1 1 -293035 Japan 

(Number) (Country) 
(B«) 



(Number) (Country) 
<#*) (B*) 

fttt, 13 5S*ItJ4l 1 9* (e) JilcSVNTTfE©*!! 



I hereby claim foreign priority under Title 35, United States Code, 
Section 119 (a)-(d) or 365(b) of any foreign application(s) for patent 
or inventor's certificate, or 365(a) of any PCT international 
application which designated at least one country other than the 
United States, listed below and have also identified below, by 
checking the box, any foreign application for patent or inventor's 
certificate, or PCT International application having a filing date before 
that of the application on which priority is claimed. 

Priority Not Claimed 

14/Onf.ober/l 999 U 

(Day/Month/Year Filed) 

(W«¥JB) 



(Day/Month/Year Filed) 

wr¥r h) 

I hereby claim the benefit under Title 35, United States Code, 
Section 119(e) of any United States provisional application(s) listed 
below. 



(Application No.) (Filing Date) 

(ttHBS*) (UJSSB) 

&jft3 6 5* (c) CS-f <«BiJSrri(::±!iL*i- 0 *fc, # 



(Application No.) (Filing Date) 

(fflH##) (W«H) 

I hereby claim the benefit under Title 35, United States Code, 
Section 120 of any United States application(s), or 365(c) of any 
PCT international application designating the United States, listed 
below and, insofar as the subject matter of each of the claims of this 
application is not disclosed in the prior United States or PCT 
International application in the manner provided by the first 
paragraph of Title 35, United States Code Section 112, I 
acknowledge the duty to disclose information which is material to 
patentability as defined in Title 37, Code of Federal Regulations, 
Section 1 .56 which became available between the filing date of the 
prior application and the national or PCT international filing date of 
application. 



(Application No.) 
(WS#-§-) 



(Application No.) 



(Filing Date) 
(BIB) 



(Filing Date) 
(WPB) 



mi, fii#©*nmc*-f ^x^nm^xmm ft ?mm 
■r < m m &±xmmx$>z tfttrvNsrt, $bi<i &:Mi^ ft 

nam. mmht'. x\tmm^^Mz.w^<D^mm^t> 



(Status: Patented, Pending, Abandoned) 

(Status: Patented, Pending, Abandoned) 

I hereby declare that all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may 
jeopardize the validity of the application or any patent issued 
thereon. 
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Japanese Language Declaration 



POWER OF ATTORNEY: As a named inventor, I hereby appoint 
the following attorney(s) and/or agent(s) to prosecute this application 
and transact all business in the Patent and Trademark Office 
connected therewith (list name and registration number) 



Donald R. Antonelli, Reg. No. 20,296; David T. Terry, Reg. No. 
20,178; Melvin Kraus, Reg. No. 22,466; William I. Solomon, Reg. 
No. 28,565; Gregory E. Montone, Reg. No. 28,141; Ronald J. 
Shore, Reg. No. 28,577; Donald E. Stout, Reg. No. 26,422; Alan E. 
Schiavelli, Reg. No. 32,087; James N. Dresser, Reg. No. 22,973 
and Carl I. Brundidge, Reg. No. 29,621 



Send Correspondence to: 

Antonelli, Terry, Stout S Kraus, LLP 
Suite 1800 

1300 North Seventeenth Street 
Arlington, Virginia 22209 



fflgmiSiS&Sfc : (B:&RX$ms£^-%) Direct Telephone Calls to: (name and telephone number) 



Telephone: (703) 312-6600 
Fax:(703)312-6666 







Full name of sole or first inventor 

Nobuaki FUJIMURA 






Inventor's signature Date 

hlo^udt, Fu.\(*Luto fR/Sep}, /2oQO 


&m 




Residence 

Kokubunji, Japan 


mm 




Citizenship 
Japan 






Post Office Address 

c/o Hitachi Denshi Kabushiki Kaisha 
Intellectual Property Dept. 32, Miyukicho, 
Kodaira-shi, Tokyo 187-8511, Japan 



(B— ^^©^if^K^fco^T^l^itl'fEfcl,, H££"f-5 (Supply similar information and signature for second and 
^ ^ ) subsequent joint inventors.) 
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H~#I33SW* 




Full name of second joint inventor, if any 
Makoto SUZUKI 






Second inventor's signature Date 

Mahrih S'cLza.ki /6/Sepf /zoco 


&m 




Residence 

Higashimurayama, Japan 


mm 




Citizenship 

Japan 






Post Office Address 

c/o Hitachi Denshi Kabushiki Kaisha 
Intellectual Property Dept. 32, Miyukicho, 
Kodaira-shi, Tokyo 187-8511, Japan 






Full name of third joint inventor, if any 

Katsumasa UENO 




Btt 


Third inventor's signature Date 
(Cai^LLMCt^ U&HD 18 /Seff.Uooo 






Residence 

Kodaira, Japan 


SIS 




Citizenship 

Japan 






Post Office Address 

c/o Hitachi Denshi Kabushiki Kaisha 
Intellectual Property Dept. 32, Miyukicho, 
Kodaira-shi, Tokyo 187-8511, Japan 






Full name of fourth joint inventor, if any 






Fourth inventor's signature Date 






Residence 






Citizenship 






Post Office Address 












Fifth inventor's signature Date 






Residence 






Citizenship 


*»# 




Post Office Address 



J 
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